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14.  ABSTRACT 

The  purpose  of  this  research  is  to  improve  stair  descent  for  lower  limb  amputees  by  providing 
sensory  feedback  of  foot  placement.  An  increasing  number  of  amputees  are  receiving  a  nerve 
transfer  surgery.  Targeted  Reinnervation,  that  can  have  profound  sensory  effects.  Touches  at 
the  site  of  the  surgery  can  feel  like  they  are  originating  from  the  amputated  limb.  This 
capability  is  an  unprecedented  opportunity  to  provide  sensory  feedback  that  is  intuitive  and 
useful,  but  sensory  recovery  after  the  surgery  is  not  well  understood.  Therefore  the  two 
Specific  Aims  of  this  project  are  to  (1)  Systematically  map  and  characterize  the  sensory 
capabilities  of  lower  extremity  Targeted  Reinnervation  (TR)  sites  under  tactile  stimulation, 
and  (2)  Measure  the  effects  of  vibrotactile  cues  of  foot  placement  on  stair  descent  of 
transtibial  amputees.  We  have  created  the  tactile  stimulation  apparatus  and  protocols  for 
both  of  these  aims,  including  a  novel  stimulator  design  and  a  novel  methodology  for 
understanding  visual-tactile  integration.  We  have  recruited  all  personnel,  received  approval 
to  conduct  the  studies,  pending  final  review  by  HRPO,  and  have  conducted  preliminary  studies 
using  control  participants  for  the  already-approved  sensory  studies. 
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Introduction 

We  seek  to  improve  stair  descent  for  lower  limb  amputees  by  providing  sensory  feedback  of  foot 
placement.  An  increasing  number  of  amputees  are  receiving  a  nerve  transfer  surgery.  Targeted 
Reinnervation,  that  can  have  profound  sensory  effects.  Touches  at  the  site  of  the  surgery  can  feel  like 
they  are  originating  from  the  amputated  limb.  This  capability  is  an  unprecedented  opportunity  to 
provide  sensory  feedback  that  is  intuitive  and  useful,  but  sensory  recovery  after  the  surgery  is  not  well 
understood. 

Therefore  the  two  Specific  Aims  of  this  project  are  to  (1)  Systematically  map  and  characterize  the 
sensory  capabilities  of  lower  extremity  Targeted  Reinnervation  (TR)  sites  under  tactile  stimulation,  and 
(2)  Measure  the  effects  of  vibrotactile  cues  of  foot  placement  on  stair  descent  of  transtibial  amputees. 

In  this  first  year  we  have  taken  the  first  steps  toward  these  aims. 
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Accomplishments 

What  are  the  major  goals  of  the  project? 

Specific  Aim  1  is  to  systematically  map  and  characterize 
the  sensory  capabilities  of  lower  extremity  Targeted 
Reinnervation  (TR)  sites  under  tactile  stimulation.  This 
aim  is  divided  into  two  Major  Tasks: 

Major  Task  1:  Development  of  Lower-Limb  Vibrotactile 
Stimulation  Technologies 


Figure  1:  Tactile  stimulation  wand.  The 
design  allows  for  careful  measurement 
and  control  of  force  and  frequency. 


We  are  developing  devices  and  techniques  for  tactile  stimulation  of  the 
residual  limb  for  amputees  who  have  not  had  TR,  and  the  TR  surgery  site  for 
those  who  have. 

There  are  two  main  technologies  under  development  for  assessing  sensory 
capabilities.  The  first  technology  (Figure  1)  is  a  single  stimulator  that  can 
precisely  measure  and  control  the  force  applied,  as  well  as  the  movement 
frequency  and  linear  displacement  of  the  skin-tactor  interface.  The  design 
incorporates  two  major  elements  -  a  constant-force  sliding  handle,  and  an 
effector  that  can  be  either  conventional  stimulators,  or  the  cam-follower 
vibrotactor  design  pictured  in  Figure  2.  The  cam-follower  design  allows  the 
frequency  to  be  set  by  commands  to  a  motor,  and  the  amplitude  to  be 
specified  by  the  profile  of  the  cam.  Conventional  vibrotactors  do  not  allow 
this  free  setting  of  both  parameters,  and  are  wider  and  flatter,  which 
reduces  the  maximum  resolution  of  an  array  of  tightly  placed  tactors. 
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Figure  2:  The  cam- 
follower  vibrotactor 

enables  a  higher- 
resolution  tactile  display 
than  standard 

vibrotactors. 
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The  second  technology  is  an  array  of  stimulators.  This  allows 
different  places  on  the  skin  to  be  stimulated  without  manually 
moving  the  tactor,  and  will  also  be  used  in  Specific  Aim  2,  when 
delivering  the  tactile  feedback  of  foot  placement.  We  are 
focusing  on  two  prototype  designs  for  this  technology,  one 
based  on  the  novel  cam-follower  vibrotactor,  and  the  other 
using  piezoelectric  vibrotactors  (Figure  3). 

We  are  also  developing  two  techniques  for  assessing  sensory 
capabilities.  The  first  is  a  protocol  of  stimulation  and  a  touch¬ 
screen  program  used  by  the  participant  to  indicate  the  felt 
sensations.  The  protocol  is  based  on  standard  psychophysical 
experimental  technique,  wherein  a  stimulation,  from  either 
monofilaments  or  vibrotactors,  is  repeated  at  the  same 
intensity  for  two  out  of  three  touches.  The  participant  indicates 
whether  and  where  the  sensation  was  felt  using  the  computer 
program  (see  Figure  4).  Correct  "whether"  answers  are  used  to 
measure  minimum  perceivable  stimulation  intensity  thresholds, 
and  "where"  answers  are  used  to  map  the  touched  locations  to 
perceived  locations.  This  protocol  has  been  conducted  for 


Figure  3:  Above:  Array  of 
piezoelectric  vibrotactors,  under 
custom  skin  interface  enclosures. 
The  strap  affixes  the  array  to  the 
thigh,  outside  of  the  socket.  Below: 
High-resolution  arrays  of  factors. 


participants  without  amputation  to  establish  baselines 
and  guide  development. 


The  second  technique  makes  use  of  Virtual  Reality  (VR)  to 
provide  visual  and  tactile  touch  stimulation,  as  depicted  in 
Figure  5.  This  system  provides  full  control  over  the  visual 
experience,  by  changing  the  virtual  body  and  virtual 
world.  This  allows  us  to  understand  how  multisensory 
experiences  of  vision  and  taction  are  combined  to  create 
useful  sensations.  By  depicting  an  intact  limb  "in  virtuo"  to 
the  participants  with  amputation  and  who  have  had  TR 
surgery,  we  are  able  to  systematically  vary  stimulation 
parameters  and  collect  quantitative  behavioral  responses. 
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Figure  4:  Touch-screen  program  for 
indicating  perceived  touches. 


Major  Task  2:  Mapping  and  characterization  studies  of  targeted  reinnervation  (TR)  sites 
This  second  major  task  for  Specific  Aim  1  is  to  conduct  the  sensory  studies  using  the  developed 
apparatus  and  techniques.  Firstly,  we  have  designed  the  studies  and  obtained  local  Veterans  Affairs 
Institutional  Review  Board  (IRB)  and  Fluman  Research  Protection  Office  (FIRPO)  approval  to  conduct 
them.  IRB  approval  within  the  VA  is  more  elaborate  than  within  the  university  setting,  and  our  center 
has  dedicated  staff  to  facilitate  approval.  We  submitted  an  initial  protocol  for  review  that  included  only 
the  sensory  mapping  and  characterization  (not  the  stair  descent)  on  3/19/2015  and  received  local  IRB 
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IRB  SUBMISSION  TRACKING  LOG 

American  Lake  &  Seattle 

Investigator:  Eric  Rombokas,  PhD  MIRB  #:  00807 

Study  Site:  Seattle  VA  Sponsor:  Dept  of  Defense 


‘Retain  complete  copies  of  all  items  submitted  or  received  pertaining  to  the  submission 


Reason  for  Submission  /  Document(s) 
Submitted* 

Date 

Submitted 
to  IRB 

Date  IRB/R&D 
Approved 

Date  Returned 

Approval 
Letter(s)  in 
files 

Comments 

IRQ  and  R&D  application 

3/19/2015 

5/12/2015 

5/27/2015 

yes 

HRPO/DoD  application 

6/3/2015 

10/13/2015 

10/13/2015 

Yes 

Initial  review  packet;  approval  forwarded  to  VA  IRB  10/14/15 

PRAF  1 

1/18/2016 

1/21/2016 

1/27/2016 

Yes 

Add  UW  IRB  to  consent  form  (now  on  V2  1/15/2016)  and  HIPAA  form 
(nowon  V2;  1/15/2016) 

CRQ  2016 

2/17/2016 

3/16/2016 

3/22/2016 

Yes 

Emailed  a  copy  of  submission  packet  and  approval  letters  to  PI  to  send 
to  DoD  from  his  UW  email  (VA  email  down  3/22/2016) 

HRPO/DoD  CRQ 

3/24/2016 

N/A 

5/20/2016 

Yes 

Emailed  copy  of  CRQ  packet  and  approval  to  HRPO;  had  to  resubmit  on 
3/28/16 

Study  Staff  Update 

7/14/2016 

7/14/2016 

7/14/2016 

Yes 

remove  Vijeth  Rai,  add  Lalit  Palve,  David  Caballero,  and  Astrini  Sie 

PRAF  2 

9/28/2016 

10/12/2016 

10/13/2016 

Yes 

add  stair  procedures,  complete  study  protocol,  add  pilot  study 
component;  revise  recruitment/screening  materials,  consent  form, 

HIPAA,  variable  list 

HRPO/DoD  PRAF  2 

10/14/2016 

Sent  via  gmail  -  attachment  too  large  for  VA  email 

Table  1:  Human  subjects  research  oversight  submissions  and  approvals 

approval  on  5/12/2015.  We  received  HRPO  approval  four  months  later.  The  protocol  was  updated  to 
include  stair  descent,  approved  10/12/2016  by  local  IRB  and  pending  approval  by  HRPO.  In  summary,  we 
are  ready,  in  technology,  protocols,  and  institutional  hurdles,  to  recruit  participants  with  amputation 
and  the  targeted  reinnervation  surgery  as  soon  as  we  receive  approval  from  HRPO. 

Mapping  and  characterization  studies  have  been  tuned  and  have  begun  for  two  control  subjects  having 
no  amputation.  Based  on  these  experiments  we  are  ready  to  begin  recruitment  of  amputees  with  and 
without  the  nerve  transfer  surgery.  Light  touch  monofilament  experiments  have  established  a  baseline 
for  accuracy  and  variance  of  touch  perception  for  relevant  anatomical  points.  A  representative  example, 
including  the  touch-screen  GUI,  is  shown  in  Figure  4.  We  have  also  found  that  simultaneous  visuo-tactile 
experience  of  touch  on  the  upper  leg  is  surprisingly  insensitive  to  mismatch  between  the  location  of 
real-world  tactile  touch  and  the  location  of  the  apparent  visual  touch.  We  placed  participants  (with  no 
amputation)  in  a  virtual  seated  position,  with  the  avatar's  leg  matching  the  dimensions  and  orientation 
of  their  real-world  leg.  A  virtual  wand  appeared  visually  to  touch  the  leg,  while  the  real-world  wand 
provided  tactile  touch,  but  offset  by  varying  degree.  Figure  7  shows  the  average  confidence  that  the 
vision  matched  the  tactile  sensation,  as  a  function  of  offset.  Even  with  6  centimeters  difference  between 
visual  and  tactile  stimulation,  subjects  responded  positively  about  half  the  time,  indicating  a  unified 
perceptual  experience  of  a  single  touch.  This  technique  will  be  used  to  understand  and  numerically 
quantify  the  sensory  capabilities  and  the  mapping  from  visual  and  tactile  touch  stimuli  to  the  resultant 
perceived  touches. 

Specific  Aim  2  is  to  determine  the  effects  of  vibrotactile  cues  of  foot  placement  on  stair  descent  of 
transtibial  amputees.  This  aim  consists  of  two  Major  Tasks: 


4 


Major  Task  3  is  the  development  of  a  speed- 
adapting  stair  descent  machine.  This  device  is  based 
on  a  commercially  available  stair  exercise  product, 
but  modified  to  operate  in  descent  mode  (opposite 
the  normal  direction  of  an  exercise  stairmill)  and  to 
sense  the  location  of  the  user  in  order  to  adapt  the 
speed  of  operation.  The  commercially  available 
device  normally  uses  electromechanical  resistance 
via  a  transmission  system  to  the  steps,  intended  for 
use  in  stair  ascent.  Our  design  calls  for  replacing  the 
alternator  with  a  powered  motor  for  reverse 
operation  (Figure  7).  We  are  now 

developing  embedded  systems  for  - 

integrating  the  motor  driver  with  a 
system  for  tracking  the  user  and 

uu* 

computing  an  appropriate  control  for 
speed  adaptation.  We  are  also 
beginning  construction  of  safety 
features  such  as  emergency  stop, 
platforms  and  handrails. 


Figure  5:  Participant's  point  of  view  in  VR 
(left)  and  real  world  (right  ) 
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Figure  6:  Confidence  for  offsets  presented  to  Site  1 
(closest  to  the  knee)  on  the  right  in  blue,  and  Site  2  to 
the  left  in  yellow. 


Major  Task  4:  Functional  study  of 
feedback  for  stair  descent.  Functional 
"dress  rehearsal"  tests  have  been 
conducted  for  real-time  feedback 
system  using  an  insole  affixed  to  a  medical 
boot.  This  allows  testing  with  people  without 
amputation,  but  simulates  the  lack  of  ankle 
dorsiflexion  and  tactile  sensation  that 
contribute  to  difficulty  in  stair  descent.  The 
insole  is  instrumented  with  force  sensors 
linearly  spaced  heel  to  toe  (Figure  8).  We  have 
developed  two  stimulation  paradigms, 
consisting  of  a  straight  "sensory  pass-through" 
where  sensor  activity  is  rendered  faithfully  to 
the  tactors  and  "placement  indicator"  where 
sensor  data  is  used  to  determine  stair  edge 
location  and  that  alone  is  rendered  to  the 
tactors.  Pending  final  approval  formal 

recruitment  will  begin,  using  fixed  stairs  and  steps.  After  speed  adapting  treadmill  is  completed  it  will  be 
incorporated,  and  is  already  part  of  the  IRB  approval. 


Figure  7:  Stair  Descent  Machine.  We  have 
replaced  the  resistive  alternator  with  a  powered 
motor  for  reverse  direction  operation. 
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Opportunities  for  training  and  dissemination  of 
results 

The  activities  in  this  report  are  broad  and 
interdisciplinary.  Research  assistants  from  Depts.  Of 
Mechanical  and  Electrical  engineering  have  spent 
extensive  time  working  with  the  investigators  to  achieve 
these  technological  and  methodological  goals.  Three 
presentations  were  given  at  the  Northwest  Biomechanics 
Symposium  2016,  and  four  manuscripts  are  in 
preparation  to  disseminate  the  results  of  these  efforts. 

We  also  occupied  a  booth  for  the  University  of 
Washington  Engineering  Discovery  Days,  which  is  an 
outreach  event  for  middle  school  to  high  school 
students,  in  which  we  demonstrated  the  prostheses, 
sensors,  and  tactors,  and  described  the  research. 

Plans  for  next  reporting  period 

The  immediate  next  phase  of  this  research  is  to  recruit  a 
cohort  of  participants  for  the  full  study,  including  sensory 
characterization  and  mapping,  to  step  tests  with  the 
tactors,  to  full  stair  ambulation.  Particular  effort  will  be 
made  to  establish  best  practices  for  delivering  tactile 
stimulation  appropriately  to  TR  receipients  based  on  the 
sensory  characterization. 

Impact 

Impact  on  the  principal  discipline  of  the  project 

We  have  conducted  the  first  "visuo  tactile  two-point  discrimination  test."  In  the  study  of  tactile  sensory 
capabilities,  a  common  test  is  to  see  how  close  two  touches  must  be  before  they  feel  like  only  one 
touch.  Using  the  virtual  reality  setup  (see  Figure  5)  we  have  created  an  equivalent  multisensory  test,  to 
understand  how  vision  and  tactile  sensation  combine  into  a  unified  feeling.  We  anticipate  this  sort  of 
multisensory  study  to  become  more  important  as  we  increasingly  use  mechanical  devices,  like  powered 
prosthetic  limbs  that  provide  sensory  feedback,  as  a  part  of  our  bodies.  Finally,  we  anticipate  the  TR 
sensory  mapping  and  characterization  study  to  provide  a  valuable  first  look  at  the  sensory  consequences 
of  lower  extremity  TR,  and  also  the  first  to  follow  recipients  longitudinally  for  months  as  they  recover 
after  surgery. 

Impact  on  other  disciplines 

Nothing  to  report. 

Impact  on  technology  transfer 

Though  it  remains  early,  we  anticipate  that  the  novel  tactor  designs  will  yield  eventual  commercial  or 
clinical  devices. 


pictured  with  exposed  sensors  but 
they  are  typically  embedded  inside. 
The  wearable  tactors  are  pictured  in 
Figure  3. 
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Impact  on  society 

Nothing  to  report. 

Changes  /  Problems 

Changes  in  approach 

Nothing  to  report. 

Problems  or  delays 

We  have  a  complex  experimental  protocol  that  has  taken  us  longer  than  we  would  like  to  have  approved 
by  IRBs  and  HRPO  (about  125%  of  expected  time).  These  approvals  are  now  in  place,  except  final  review 
by  HRPO,  and  we  do  not  anticipate  further  action  for  resolution. 

Impacts  on  expenditures 

Funding  for  the  project  was  received  too  late  to  recruit  research  assistants  for  the  normal  academic 
year,  so  they  were  recruited  in  winter  instead  of  fall.  Therefore  there  is  an  underexpenditure  on  head 
count  for  the  first  year.  At  this  time  the  project  is  fully  staffed  and  we  intend  to  use  these  funds  as 
needed  to  support  further  personnel  for  the  remainder  of  the  project  period. 

Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  or  select 
agents 

Nothing  to  report. 

Products 

Publications,  conference  papers,  and  presentations 

Three  presentations  were  given  at  the  Northwest  Biomechanics  Symposium  2016 

Technologies  or  techniques 

We  have  pioneered  a  VR  multisensory  fusion  protocol  that  we  will  be  disseminating  after  further 
experimentation.  We  are  also  designing  a  variety  of  potentially  impactful  mechanical  designs  for  tactile 
stimulators. 
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VR  sensory  protocol  and  apparatus 

Funding  Support: 

Changes  in  the  active  other  support  of  the  PD/PI(s)  or  senior/key  personnel  since  the  last 
reporting  period 
Nothing  to  report. 

Other  organizations  involved  as  partners 
University  of  Washington 
Seattle,  WA 

Collaboration,  Personnel  Exchanges 

Harborview  Medical  Center 
Seattle,  WA 
Collaboration 

Special  Reporting  Requirements 

Quad  chart 
See  attachments 

Appendices 

None 


Sensory  Feedback  for  Lower  Extremity  Incorporating  TMR 

MR140172  Neuromusculoskeletal  Injuries  Research  Award 
Funding  Opportunity  Number:  W81 XWH-1 4-DMRDP-CRMRP-NMSIR 

PI:  Eric  Rombokas  Org:  Seattle  Institute  for  Biomedical  and  Clinical  Research  Award  Amount:  1.5M 


Study  Aims 

*  Map  and  characterize  the  sensory  capabilities  of  lower  extremity 

Targeted  Reinnervation  (TR)  sites  under  vibrotactile  stimulation. 

*  Measure  the  effects  of  vibrotactile  cues  of  foot  placement  on  stair 

descent  of  transtibial  amputees. 

Approach 

Assess  sensory  consequences  ofTR  in  lower  extremity  via  Semmes- 
Weinstein  monofilament  exam,  then  use  hand-held  vibrotactile 
stimulator  to  measure  for  the  vibrotactile  haptic  modality  that  would 
actually  be  used  in  an  integrated  sensorized  prosthetic  system. 

Measure  the  effects  of  providing  vibrotactile  feedback  of  foot  placement 
on  self-selected  speed  of  transtibial  amputees  performing  stair 
descent.  Subjects  will  descend  integrated  motion-capture  speed- 
adaptive  escalator. 


Timeline  and  Cost 


Activities  CY 

16 

17 

18 

19 

Develop  Vibrotactile  Actuators 

Develop  automatic  stair  machine 

a 

Sensory  mapping  of  TR  sites 

■ 

Stair  Descent  with  feedback 

i 

Estimated  Budget  ($K) 

$496 

$498 

$497 

$000 

Vibrotactor  single  stimulator  (left,  top)  and  worn  array  (left,  bottom).  Vibrotactile 
sensory  feedback  can  deliver  sensation  of  forces  and  foot  events  to  the  lower 
extremity  amputee.  Users  having  targeted  reinnervation  feel  these  sensations  as  if 
they  are  originating  at  the  absent  limb.  Participants  indicate  where  they  felt 
sensations  (middle).  Sensory  characterization  is  also  performed  by  providing 
simultaneous  visual  and  tactile  sensation  (right). 


Goals/Milestones 

-  Speed-adapting  stairmill  user  tracking  complete 

-  Final  HRPO  approvals  for  stair  descent  protocol 

CY16  Goals  - 

-  Functionality  and  safety  tests  of  integrated  motion  capture  stair 
machine 

-  Functionality  tests  of  Handheld  Feedback  Wand  and  Wearable 
Feedback  Array 

-  Sensory  Mapping  /  Characterization  and  Stair  Descent 

-  Gait  lab  tests  of  stair  descent  for  amputees  without  TR  surgery 

-  Characterization  of  TR  site  sensory  capabilities 
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